Revalidate the cache files on every request (0.3.2) - #24

Summary

HttpCacheStream.request() now revalidates the cache files against the running download on every call. A cache whose files were deleted or changed underneath it (by Android's cache management, say) is reset, and the request waits for its range to download again instead of its response failing. The deletion check #23 added to request() moves into _cacheFileState(), which now runs it for every refresh. This is release 0.3.2: the pubspec is bumped and the CHANGELOG's Unreleased section becomes 0.3.2. Nothing is published.

How it works

• _cacheFileState() checks the partial file against the running download: the file must hold at least the bytes the download has saved (filePosition, read before the stat). A deleted or truncated file fails that check. So do the existing ones: a complete file of the wrong size, or a partial one longer than the source. Each throws InvalidCacheSizeException, and refreshCacheState() cancels the download it judged while still holding the file lock, then resets the cache.
• While a download writes, the state takes the download's position and source length. That position is ahead of the file by the bytes not yet saved, and the saved bytes go in the new CacheState.filePosition. Once the download stops writing, the state is read from the files, and only then is the partial file renamed to complete.
• A reset while the download is writing cancels it and waits; the download ends on the exception and resets the cache. A reset after it stopped writing resets directly, without waiting for it, since the download's own completion refreshes the state and would otherwise wait on the reset.

Bugs fixed along the way

• Corrupt cache after a resume. A resumed download whose partial file was deleted, truncated or written to before its first write appended at the wrong offset (appending re-creates a deleted file). If the download stopped before anything noticed, the next resume completed a cache file of the right size with the wrong content. BufferedIOSink now checks, when it reopens a file to append, that the file ends where the sink resumes it. A file that doesn't is an invalid cache, which resets. The new test fails on main with a hash mismatch.
• resetCache() ignored between responses. A download waiting for the origin, or waiting to retry, had no open response for the reset's exception to end. It returned as if stopped and resumed the old partial cache. CacheDownloader now ends on an invalid-cache cancel wherever it arrives (between responses, while saving its last bytes, or from a failed save) and serves no more requests after it. The new test fails on main: the download resumes from the old partial.
• Position dropped to 0. A refresh during a download of unknown length (a chunked response) reported position 0, and refreshing on every request would have done that constantly. The new test fails against main's stream code.

Behaviour changes

• CacheState gains filePosition. It is optional on CacheState.incomplete and defaults to position, and incomplete states compare by it too.
• Every request() does one or two stat calls under the file lock.
• A complete cache file of the wrong size now resets the whole cache, including any partial file next to it.

Limits

• Truncation while the download keeps writing is caught only until the download writes past the cut. Dart's append mode seeks to the end once, so later writes leave zeros behind. Deletion is always caught.
• Not tested on Windows. The rename waits until the download has released the file, which is what Windows needs.

Tests

• test/e2e/cache_files_changed_test.dart (replaces partial_deleted_test.dart):
  • partial file deleted mid-download
  • concurrent requests reset the cache once
  • resumed download whose partial file was deleted before it wrote
  • the same with the rest of the body arriving at once
  • complete file of the wrong size
  • a refresh keeps an unknown-length download's position
• test/e2e/reset_cache_test.dart: resetCache() while the download waits for the origin.
• test/io/buffered_io_sink_test.dart: resuming a deleted, truncated or written-to file fails without writing to it.
• test/unit/cache_state_test.dart: filePosition defaults and equality.
• Results:
  • flutter test: 156 pass, in 3 runs.
  • dart analyze lib test: no issues.
  • flutter analyze: only the existing issues in benchmarker/.

trail check flags three lines (cache_downloader.dart:168, http_cache_stream.dart:380, cache_files_changed_test.dart:57) as bringing back what the learning saved for this work ruled out. They are the fixes that learning records (the block-bodied flush error handler, the cancel under the lock), not the versions it ruled out.